Some imports:
In [ ]:
%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
try:
import seaborn
except:
pass
pd.options.display.max_rows = 8
In [ ]:
from IPython.display import HTML
HTML('<iframe src=http://www.eea.europa.eu/data-and-maps/data/airbase-the-european-air-quality-database-8#tab-data-by-country width=900 height=350></iframe>')
I downloaded and preprocessed some of the data (python-airbase): data/airbase_data.csv. This file includes the hourly concentrations of NO2 for 4 different measurement stations:
Import the csv file:
In [ ]:
!head -5 data/airbase_data.csv
As you can see, the missing values are indicated by -9999. This can be recognized by read_csv by passing the na_values keyword:
In [ ]:
data = pd.read_csv('data/airbase_data.csv', index_col=0, parse_dates=True, na_values=[-9999])
Some useful methods:
head and tail
In [ ]:
data.head(3)
In [ ]:
data.tail()
info()
In [ ]:
data.info()
Getting some basic summary statistics about the data with describe:
In [ ]:
data.describe()
Quickly visualizing the data
In [ ]:
data.plot(kind='box', ylim=[0,250])
In [ ]:
data['BETR801'].plot(kind='hist', bins=50)
In [ ]:
data.plot(figsize=(12,6))
This does not say too much ..
We can select part of the data (eg the latest 500 data points):
In [ ]:
data[-500:].plot(figsize=(12,6))
Or we can use some more advanced time series features -> next section!
When we ensure the DataFrame has a DatetimeIndex, time-series related functionality becomes available:
In [ ]:
data.index
Indexing a time series works with strings:
In [ ]:
data["2010-01-01 09:00": "2010-01-01 12:00"]
A nice feature is "partial string" indexing, where we can do implicit slicing by providing a partial datetime string.
E.g. all data of 2012:
In [ ]:
data['2012']
Normally you would expect this to access a column named '2012', but as for a DatetimeIndex, pandas also tries to interprete it as a datetime slice.
Or all data of January up to March 2012:
In [ ]:
data['2012-01':'2012-03']
Time and date components can be accessed from the index:
In [ ]:
data.index.hour
In [ ]:
data.index.year
In [ ]:
data = data['1999':]
In [ ]:
data[data.index.month == 1]
In [ ]:
data['months'] = data.index.month
data[data['months'].isin([1, 2, 3])]
In [ ]:
data[(data.index.hour >= 8) & (data.index.hour < 20)]
In [ ]:
data.between_time('08:00', '20:00')
A very powerfull method is resample: converting the frequency of the time series (e.g. from hourly to daily data).
The time series has a frequency of 1 hour. I want to change this to daily:
In [ ]:
data.resample('D').head()
By default, resample takes the mean as aggregation function, but other methods can also be specified:
In [ ]:
data.resample('D', how='max').head()
The string to specify the new time frequency: http://pandas.pydata.org/pandas-docs/dev/timeseries.html#offset-aliases
These strings can also be combined with numbers, eg '10D'.
Further exploring the data:
In [ ]:
data.resample('M').plot() # 'A'
In [ ]:
# data['2012'].resample('D').plot()
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
resample can actually be seen as a specific kind of groupby. E.g. taking annual means with data.resample('A', 'mean') is equivalent to data.groupby(data.index.year).mean() (only the result of resample still has a DatetimeIndex).
In [ ]:
data.groupby(data.index.year).mean().plot()
But, groupby is more flexible and can also do resamples that do not result in a new continuous time series, e.g. by grouping by the hour of the day to get the diurnal cycle.
1. add a column to the dataframe that indicates the month (integer value of 1 to 12):
In [ ]:
2. Now, we can calculate the mean of each month over the different years:
In [ ]:
3. plot the typical monthly profile of the different stations:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
Add a column indicating week/weekend
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]:
Tip: the boxplot method of a DataFrame expects the data for the different boxes in different columns)
In [ ]:
In [ ]:
In [ ]:
In [ ]:
In [ ]: